DataHub 3 Phase 3 target architecture and reference implementation plan

Problem summary

The current DMC DataHub integration is partly implemented in obsolete architecture and spreads integration concerns across Domain, Application, and Infrastructure.

Current observations from the codebase:

• The active host loads DataHub settings from src\Clients\EnergyConnect.WebApi\Settings\datahub.json and already supports Microsoft.FeatureManagement and Azure App Configuration feature flags.

• Legacy/current transport implementations live in DMC.Infrastructure\Services\DataHub.

• Domain still owns some integration-facing abstractions and DTOs, which is not the desired target.

• DataHubInboxDocumentHandling mixes transport parsing, version branching, mapping, and business handling.

• IRsmMessageHandler<TMessage> shows a promising inbound shape, but the surrounding document model structure and placement are not yet target architecture.

The goal for DataHub 3 Phase 3 is not to extend that legacy design. It is to create a blank-page target architecture and reference implementation in DMC.Infrastructure\Integrations\DataHub, implemented as new code beside the current solution, so it can later become the migration target for the rest of the integration.

Target architecture

Architectural intent

DataHub 3 Phase 3 should be implemented as a clean new integration stack under Infrastructure, but not as a dedicated Phase3 namespace or folder.

• Infrastructure owns all DataHub-specific concerns:

  - generated document contracts

  - JSON schemas

  - serialization

  - authentication

  - HTTP clients

  - routing to DataHub endpoints

  - integration-specific inbound handlers

  - integration-specific outbound publishers

  - protocol-specific diagnostics

• Application owns only business use cases:

  - commands/queries and typed handlers in existing business namespaces

  - feature-local event handlers that trigger outbound integrations

  - business orchestration and transaction boundaries

• Domain owns only business truth:

  - entities

  - value objects

  - invariants

  - domain events

Core rules

1. Do not create a DMC.Application.DataHub area.

2. Do not implement ICommandHandler<> or IQueryHandler<> inside the integration layer.

3. Incoming DataHub documents are handled by custom integration handlers in Infrastructure, which map to application commands/queries and dispatch those via the existing application dispatcher interfaces.

4. Outgoing DataHub documents are triggered from events, not from direct application service calls.

5. Generated DataHub models never enter Domain.

6. Generated DataHub models should stay colocated with their schema definitions instead of being split into separate Schemas and Generated folder trees.

7. Phase 3 should be built as new code separately from the existing DataHub integration task, while avoiding changes to existing integration code except the minimal coexistence wiring needed to use the new implementation.

Reference implementation strategy

Blank-page reference implementation

DataHub 3 Phase 3 should be introduced as a reference implementation beside the current integration, not by gradually reshaping the old Services\DataHub code first and not by introducing a separate Phase3 namespace hierarchy.

That means:

• create a new Infrastructure integration package

• keep legacy/current implementations in place

• build the first Phase 3 integration point end-to-end in the new structure

• validate the architecture on that slice

• later migrate old integrations into the new structure during maintenance or after Phase 3 is proven

This keeps the first implementation focused on target architecture quality instead of legacy compromise.

Implementation progress

Completed so far:

1. Added the new DMC.Infrastructure\Integrations\DataHub scaffold.

2. Added simple shared feature flags in both WebApi and BatchJobs:

   - DataHubPhase3Services

   - DisableLegacyDataHub2Services

3. Consolidated the official DataHub schema set into DMC.Infrastructure\Integrations\DataHub\Schemas as the single source of truth.

4. Added business-side application commands for metering point update confirmations/rejections in the existing MeteringPoints feature namespace.

5. Removed the duplicated staged schema copies and the build-time schema normalization/generation automation after concluding that it added more complexity than value at the current stage.

6. The repository now keeps the official schema files in one place only, ready for later targeted code generation once the correct Phase 3 BRS/message mapping is validated.

7. Normalized the schema-set file:///C:/Users/Public/Documents/iec.ch/TC57/2020/ references to repo-local relative references in DMC.Infrastructure\Integrations\DataHub\Schemas, while keeping the schema files in a single canonical location.

8. Generated the first pilot contract family under DMC.Infrastructure\Integrations\DataHub\BRS\BRS023-RequestValidatedMeasureData:

   - RSM015-RequestValidatedMeasureData

   - RSM015-RejectRequestValidatedMeasureData

   - RSM012-NotifyValidatedMeasureData

9. Pivoted the pilot away from Corvus to simpler NJsonSchema-generated transport classes and removed the temporary Corvus/NSwag repo tooling again.

10. Simplified the RSM015 pilot seam so generated integration classes are internal, the redundant request/document wrapper records are removed, and only an internal factory remains.

11. Added a focused unit test for the simplified RSM015 factory and confirmed both DMC.Infrastructure and DMC.Tests build with the new pilot code in place.

Current reference BRS status:

• There is no official schema in json-skemaer-v173 whose name clearly corresponds to UpdateMasterDataMeteringPoint.

• Two plausible official schema families were identified:

  - Request/Confirm/Reject change Accounting Point characteristics

  - Request/Confirm/Reject change billing master data

• Billing master data is charge-oriented.

• Accounting Point Characteristics is the only official candidate that carries point-master-data-like fields (for example metering method, settlement method, physical status, connection type, grid areas, meter reference, capacity, address, and market evaluation point identifiers).

• However, it is not validated that Accounting Point in the DataHub 3 schema set is equivalent to the DMC MeteringPoint business concept.

• Because that equivalence is unverified, the temporary Infrastructure-to-Application mapping from AccountingPointCharacteristics confirm/reject documents to ConfirmMeteringPointUpdateCommand / RejectMeteringPointUpdateCommand was removed again.

Official schemas needed before the next implementation slice:

• Either:

  - the official DataHub 3 schema family that actually represents the DMC metering-point master-data change business flow

• Or:

  - a business confirmation that Accounting Point Characteristics is the correct DataHub 3 replacement for the current metering-point master-data update flow

These must be official DataHub schemas or a validated business mapping; locally invented placeholder schemas should not be used.

Proposed folder and project structure

Root

Recommended root:

• DMC.Infrastructure\Integrations\DataHub

Recommended top-level structure:

• Authentication

• Clients

• Common

• Diagnostics

• FeatureFlags

• Serialization

• BRS

BRS-oriented structure

The main organizing unit should be BRS / integration point, not generic layers like Schemas vs Generated.

Recommended structure:

DMC.Infrastructure\Integrations\DataHub\

  Authentication\

  Clients\

  Common\

  Diagnostics\

  FeatureFlags\

  Serialization\

  BRS\

    BRS-xxx\

      Inbound\

        <DocumentName>.schema.json

        <DocumentName>.schema.g.cs

        <DocumentName>Mapper.cs

        <DocumentName>Handler.cs

      Outbound\

        <DocumentName>.schema.json

        <DocumentName>.schema.g.cs

        <DocumentName>Factory.cs

        <DocumentName>Publisher.cs

Why this structure

• each integration point becomes self-contained

• schema and generated C# stay together

• inbound and outbound logic for the same integration point are easy to review together

• reference implementation can be built one BRS at a time

• later migration of existing endpoints can follow the same template

Dependency direction and responsibilities

Direction

• API -> Application

• Application -> Domain

• Infrastructure depends on Application dispatching abstractions only where needed to invoke business commands/queries

• Domain stays independent of generated DataHub documents and protocol details

Application placement rule

There should be no DataHub-specific application namespace or module.

Instead:

• business commands/queries remain in their existing feature namespaces

• outbound trigger handlers live near the owning business feature

• integration-specific document handling stays in Infrastructure

Infrastructure placement rule

Infrastructure owns the full DataHub document pipeline:

• contract generation

• schema storage

• mapping from document to application command/query

• mapping from application/domain data to document

• dispatch to application handlers where needed

• DataHub HTTP/auth concerns

Incoming message design

Target pattern

The existing IRsmMessageHandler<TMessage> idea is close to the desired target, but the target version should live in the integration layer and sit on a cleaner document hierarchy.

Recommended direction:

• define integration-local document abstractions in Infrastructure

• generated document models implement or inherit from a small integration-specific document contract

• define infrastructure-local generic handlers such as:

  - IRsmMessageHandler<TDocument>

  - IRsmInboundMessageHandler<TDocument>

  - or equivalent naming inside the DataHub integration package

Handler responsibility

An inbound handler in Infrastructure should:

1. receive a generated document type

2. perform integration-specific validation/mapping

3. build one or more application commands/queries

4. dispatch those through ICommandHandler / IQueryHandler

5. keep all DataHub-specific branching outside Application handlers

Important rule

The integration layer must not contain ICommandHandler<T> or IQueryHandler<T> implementations.

It may call the application dispatchers, but the business handlers themselves remain in Application.

Application contract for inbound processing

Inbound DataHub documents should be mapped to application commands, not to DataHub-specific application models.

That means:

• Infrastructure understands DataHub

• Application understands business use cases

• Domain understands business truth

Outgoing message design

Trigger model

Outgoing Phase 3 documents should be triggered from events.

Preferred direction:

1. domain/business activity raises a domain event

2. an application event handler in the owning business feature reacts

3. that event handler invokes the new DataHub publisher path in Infrastructure

4. Infrastructure builds the generated document, serializes it, authenticates, and sends it

Placement rule

The application event handler should be colocated with the owning feature, not collected in a DataHub namespace.

Data boundary

Outbound publishing should use business data and business events as input, then map in Infrastructure to the generated DataHub document.

Generated DataHub DTOs must not leak back into Domain.

Document model structure

Recommendation

Use a better-structured RSM document model in Infrastructure with shared abstractions for:

• common header identity

• message correlation

• payload collections

• request/response distinctions

• BRS-specific specialization

The exact inheritance tree should be designed for clarity and generated-model compatibility, but the intent is:

• shared RSM semantics are modeled once

• BRS-specific generated classes remain strongly typed

• handlers can operate generically where useful without losing specific payload types

Design preference

The generated classes should stay close to the schemas, and handwritten behavior should remain in adjacent handwritten files or partial types rather than in a separate generated-model subtree.

Generated contract strategy

Tooling

Use NJsonSchema as the preferred code generation tool.

Source layout

Do not split Schemas and Models into separate folder trees.

Instead:

• place the schema file beside the generated .g.cs output

• keep handwritten mappers/handlers next to the corresponding schema/generated model

• make generation deterministic and repeatable

Generation workflow

Recommended workflow:

1. place a schema file inside the BRS integration-point folder

2. run a deterministic generator step using NJsonSchema

3. emit <SchemaName>.g.cs in the same folder

4. keep custom code in separate handwritten files

5. never hand-edit generated output

Tooling shape to validate later

Preferred direction is either:

• a small build step / MSBuild target, or

• a source-generator-backed workflow that still emits generated files in a predictable place for review

The key requirement is that the generated C# appears logically under the schema definition in the integration-point folder structure.

Mapping boundaries

Required boundaries

The design should enforce these boundaries:

1. schema + generated document -> Infrastructure-only document model

2. Infrastructure document model -> Application command/query

3. Application handler -> Domain entities/value objects/services

For outbound:

1. domain event / business state -> Application event handler

2. Application event handler -> Infrastructure publisher call

3. Infrastructure publisher -> generated DataHub document

4. generated document -> serialization and transport

Anti-leak rules

• no generated DataHub DTOs in Domain

• no generated DataHub DTOs in business command/query contracts unless unavoidable and explicitly justified

• no business invariants inside document mappers

• no protocol-specific HTTP/auth code outside Infrastructure

Feature toggle strategy

Requirement

Use standard .NET feature toggling via Microsoft.FeatureManagement.

Recommended first flags

Use separate features for enabling the new Phase 3 implementation and disabling the old DH2 paths.

Recommended initial flags:

1. DataHubPhase3Services

2. DisableLegacyDataHub2Services

Why this split

This supports the rollout scenarios you want:

• deploy Phase 3 code without using it

• enable new Phase 3 services in test

• keep old DH2 active while Phase 3 is enabled in test

• run both in parallel in test if needed

• disable old DH2 at cutover independently of enabling Phase 3

Recommended behavior matrix

| DataHubPhase3Services | DisableLegacyDataHub2Services | Result |

|---|---|---|

| false | false | legacy/current behavior only |

| true | false | new Phase 3 enabled, old DH2 still available for coexistence / test |

| true | true | cutover mode: new Phase 3 active, old DH2 disabled |

| false | true | invalid / unsupported combination; fail fast in startup validation |

Placement

Feature evaluation logic should live in the new integration layer under FeatureFlags, while the source of truth remains the standard FeatureManagement configuration hierarchy.

Coexistence and cutover strategy

Coexistence principle

Phase 3 must be implemented separately from the existing integration and remain deployable beside it.

Cutover stages

1. build the new Phase 3 reference implementation in the new folder structure

2. ship it dark with feature flag off

3. enable Phase 3 in test with legacy DH2 still active

4. run comparison / parallel scenarios in test

5. disable legacy DH2 at cutover using feature flags

6. after stabilization, migrate additional existing integrations into the new structure

Migration after go-live

When Phase 3 is live, existing integrations can be moved gradually into the new target architecture:

• during maintenance

• or earlier if the new target architecture proves stable enough for production migrations

Parallel-run and testing strategy

Parallel-run intent

Parallel operation should mainly be a test-environment capability.

Recommended uses:

• validate the new inbound path while legacy remains available

• validate the new outbound document generation against expected payloads

• compare new and old routing/behavior before disabling legacy flows

Safety rules

• avoid duplicate externally visible business side effects in production

• prefer comparison and controlled test execution over blind double-send

• log correlation identifiers so outcomes can be compared across old and new paths

Test coverage focus

When implementation begins, cover:

1. generated schema-to-model workflow

2. inbound handler mapping to application commands

3. outbound event-triggered publisher flow

4. feature-flag combinations and startup validation

5. coexistence with legacy/current DataHub flows

6. first reference BRS integration point end-to-end

New integration tests should go in Tests\EnergyConnect.IntegrationTests.

Risks and open questions

Risks

• the old architecture may tempt reuse that weakens the blank-page reference implementation

• generated-model inheritance may be constrained by the shape of the JSON schemas and NJsonSchema output

• event-triggered outbound flows need careful scoping so business handlers do not become transport-aware

• parallel-run must avoid duplicate external side effects

Open questions

1. Do the available official schemas fully cover the selected metering point master data update flow?

2. For the first reference implementation, should outbound send be immediate from the event handler or queued through an explicit infrastructure outbox in the new integration layer?

Concrete implementation checklist

1. Create the new blank-page reference implementation under DMC.Infrastructure\Integrations\DataHub without introducing a dedicated Phase3 namespace/folder.

2. Add BRS-oriented folder structure and colocate schemas with generated .g.cs files.

3. Introduce integration-local document abstractions and generic inbound handler interfaces inspired by IRsmMessageHandler<TMessage>.

4. Implement inbound handlers in Infrastructure that map generated DataHub documents to application commands and dispatch them via ICommandHandler / IQueryHandler.

5. Introduce event-triggered outbound publishing from application event handlers in existing business namespaces.

6. Implement Phase 3 publishers/clients/auth/serialization entirely in Infrastructure.

7. Add standard Microsoft.FeatureManagement flags:

   - DataHubPhase3Services

   - DisableLegacyDataHub2Services

8. Build the selected metering point master data update flow as the first paired inbound/outbound reference implementation.

9. Add integration tests for the reference implementation, feature flags, and coexistence behavior.

10. Keep legacy/current integrations untouched except for coexistence wiring.

11. After Phase 3 is live, migrate existing integrations gradually into the new target structure.

Execution-oriented todos

1. Use metering point master data update as the first paired inbound/outbound reference BRS.

2. Keep the BRS\... folder template and generation workflow around colocated schemas and .g.cs files.

3. Define the integration-local document abstractions and generic inbound handler interface shape.

4. Implement inbound handlers once the official inbound schemas are available.

5. Implement the outbound event-triggered publisher flow once the official outbound schema is available.

6. Keep feature-flag handling simple and configuration-driven in both WebApi and BatchJobs.